The matching points cant be decided absolutely by its residuals just relying on epipolar geometry residuals, which influences the selection of optimum inlier set. So a novel fundamental matrix calculation algorithm was proposed based on three-view constraint. Firstly, the initial fundamental matrices were estimated by traditional RANdom SAmple Consensus (RANSAC) method. Then matching points existed in every view were selected, and the epipolar lines of points not in the common view were calculated in fundamental matrix estimation. Distances between the points in common view and the intersection of its matching points epipolar lines were calculated. Under judgment based on the distances, a new optimum inlier set was obtained. Finally, the M-Estimators (ME) algorithm was used to calculate the fundamental matrices based on the new optimum inlier set. Through a mass of experiments in case of mismatching and noise, the results indicate that the algorithm can effectively reduce the influence of mismatch and noise on accurate calculation of fundamental matrices. It gets better accuracy than traditional robust algorithms by limiting distance between point and epipolar line to about 0.3 pixels, in addition, an improvement in stability. So, it can be widely applied to fields such as 3D reconstruction based on image sequence and photogrammetry.